Finite Sample Convergence Rates of Zero-Order Stochastic Optimization Methods

• Let Ak denote the set of methods that observe a sequence of data pairs Y t = (F (θ, X ), F (τ , X )), 1 ≤ t ≤ k, and return an estimate θ̂(k) ∈ Θ. • Let FG denote the class of functions we want to optimize, where for each (F, P ) ∈ FG the subgradient g(θ;X) satisfies EP [‖g(θ;X)‖2∗] ≤ G. • For each A ∈ Ak and (F, P ) ∈ FG, consider the optimization gap: k(A, F, P,Θ) := f (θ̂(k))− inf θ∈Θ f (θ) = EP [ F (θ̂(k);X) ] − inf θ∈Θ EP [F (θ;X)] . • Define minimax error of zero-order optimization: k(FG,Θ) := inf A∈Ak sup (F,P )∈FG E[ k(A, F, P,Θ)].